Unraveling Students' Interaction Around a Tangible Interface Using Gesture Recognition

In this paper, we describe techniques to use multimodal learning analytics to analyze data collected around an interactive tangible learning environment. In a previous study [4], we designed and evaluated a Tangible User Interface (TUI) where dyads of students were asked to learn about the human hearing system by reconstructing it. In the current study, we present the analysis of the data collected in form of their gestures, and we describe how we extracted meaningful predictors for students’ learning from this datasets. We explored how Kinect data can inform “in-situ” interactions around a tabletop (i.e. using clustering algorithms to find prototypical body positions). We discuss the implications of those results for analyzing data from rich, multimodal learning environments.